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The  goal  of  this  project  is  to  examine  the  consequences  of  observer  motion  for 
visual  function.  The  research  has  focussed  bn  two  issues:  One  issue  is  how  a 
grossly  time- varying  retinal  input  (because  of  eye,' head,  and  body  motion)  results 
in  the  perception  of  a  continuous  and  directionally  stable  visual  world.  A  second 
issue  concerns  hew  the  information  in  successive  views  is  related,  and  the  nature 
of  the  visual  infomation  retained  from  previous  views.  Understanding  these  processes 
is  important  for  a  wide  variety  of  visuo-irotor  tasks. 


Progress  has  been  made  on  the  following  projects: 

1)  The  role  of  the  visual  scene  and  eye  position  signals  in  visual  stability. 

2)  The  role  of  attention  in  integrating  across  saccades. 

3)  Reference  frames  for  spatial  memory. 

4)  Hand-eye  coordination  during  complex  tasks. 

5)  Detectability  of  changes  during  saccades.  . .....  .......  ........... ,A 

6)  Short  term  visual  memory  of  complex  scenes. 
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The  goal  of  this  project  was  to  examine  the  consequences  of  observer  motion 
for  visual  function.  The  research  has  focused  on  two  issues:  One  issue  is  how  a 
grossly  time-varying  retinal  input  (because  of  eye,  head,  and  body  motion)  results  in 
the  perception  of  a  continuous  and  directionally  stable  visual  world.  A  second  issue 
concerns  how  the  information  in  successive  views  is  related,  and  the  nature  of  the 
visual  information  retained  from  previous  views.  Understanding  these  processes  is 
important  for  a  wide  variety  of  visuo-motor  tasks. 

In  the  grant  period  progress  has  been  made  on  the  following  projects. 

The  role  of  the  visual  scene  and  eye  position  signals  in  visual  stability. 


In  collaboration  with  Jeff  Pelz,  I  have  developed  a  novel  technique,  using  the 
perceived  movement  of  afterimages  of  complex  scenes,  to  demonstrate  that  the 
stability  of  the  visual  scene  depends  on  the  nature  of  the  visual  context.  In  the  dark, 
the  position  of  a  single  object  is  computed  using  eye  position  information.  In  the 
light,  however,  the  experiment  shows  that  stationarity  is  attributed  to  the  visual 
scene,  despite  eye  position  information  to  the  contrary.  This  technique  is  relatively 
easy  to  use  and  has  a  lot  of  potential  for  exploring  the  properties  of  the  visual  scene 
which  determine  its  relative  weight  in  the  choice  of  a  stable  reference  frame.  (This 
technique  was  described  in  the  original  proposal  Experiment  3B.)  This  work  was 
reported  at  ARVO  in  1991  and  a  manuscript  is  in  press  in  Vision  Research. 


The  role  of  attention  in  integrating  across  saccades. 


□ 


This  work  is  in  collaboration  with  Joel  Lachter,  a  graduate  student  partially  funded 
by  the  project.  It  is  a  modified  version  of  Experiment  6  in  the  proposal.  In  an  earlier  ~g* 
experiment  we  demonstrated  that  the  perception  of  form  can  be  achieved  whether  □ 
or  not  the  eye  is  stationary,  and  that  very  precise  spatial  relationships  can  be 
computed  across  different  eye  positions.  We  have  now  demonstrated  that  this 
ability  requires  attention.  This  suggests  that  only  a  sparse  representation  of  the 
visual  scene  is  maintained  across  saccades.  (This  is  consistent  with  current 
computational  'active  vision'  approaches.)  The  work  has  been  reported  at  ARVO 
1991,  and  a  manuscript  is  under  revision  for  Perception  &  Psychophysics 


aiw/or 


Reference  frames  for  spatial  memory. 


Spatial  memory  plays  a  crucial  but  relatively  unexplored  role  in  much  of  our 
motoric  interaction  with  the  world.  Previous  work  (described  in  Experiment  10  in 
the  proposal)  has  demonstrated  the  use  of  both  body  centered  and  object  centered 
reference  frames  in  encoding  the  positions  of  objects.  In  this  experiment  Keith 
Karn,  Per  Moeller  and  I  explored  whether  object  centered  reference  frames  have  the 
advantage  of  being  more  robust  across  a  series  of  changes  in  eye  position. 
Surprisingly,  both  forms  of  encoding  appeared  to  be  equally  robust.  This  has 
implications  for  the  source  and  precision  of  the  eye  position  information.  We 
estimate  that  information  about  eye  position  in  a  head  centered  reference  frame  is 
available  to  the  visual  system  with  a  standard  deviation  of  less  than  1.4  degrees. 
This  work  was  reported  at  ARVO  1991  and  a  manuscript  has  been  published  in  the 
proceedings  of  the  6th  European  Conference  on  Eye  Movements.  A  second 
manuscript  has  bee  a  submitted  to  experimental  brain  research. 


PERFORMANCE  OF  COMPLEX  TASKS  INVOLVING  HAND-EYE 
COORDINATION. 

Copying  Task  Using  Macintosh  Display. 

In  the  task  we  have  chosen,  subjects  copy  a  pattern  of  colored  blocks  on  a 
computer  screen  using  the  mouse  to  move  blocks  around  the  display.  Recent 
successful  robotic  models  of  complex  tasks  avoid  computationally  expensive 
internal  representations  by  allowing  frequent  access  to  the  sensory  input  during  the 
problem  solving  process.  These  models  use  so  called  'deictic  primitives'  in  which 
aspects  of  a  scene  can  be  referred  to  by  denoting  that  part  of  the  scene  with  a  special 
marker,  such  as  the  fixation  point.  We  have  little  knowledge  of  how  humans 
actually  perform  in  comparable  sensori-motor  tasks.  We  have  shown  so  far  that 
human  performance  is  also  characterized  by  deictic  strategies  and  limited  memory 
representations.  This  suggests  that  current  approaches  in  robotics  are  also  useful  for 
understanding  human  brain  mechanisms.  It  also  suggests  a  computational  rationale 
for  the  limitations  on  human  working  memory.  The  limited  nature  of  human 
working  has  been  taken  as  a  kind  of  explanatory  primitive  in  understanding 
cognitive  processes.  However,  there  has  been  surprisingly  little  effort  directed  at 
understanding  why  it  is  limited,  and  how  these  limitations  play  themselves  out  in 
normal  behavior.  The  'active  vision'  approach  in  robotics  forces  a  new 
consideration  of  the  computational  role  of  working  memory.  It  seems  likely  at  this 
point  that  there  is  a  real  advantage  to  be  gained  by  such  a  system  in  terms  of 
simplifying  the  underlying  cortical  decision  making  processes  and  minimizing  the 
need  for  a  central  executor.  A  better  understanding  of  how  the  system  works  as  a 
whole  should  provide  better  guidance  in  how  to  approach  the  underlying  neural 
organization.  A  manuscript  on  this  work  is  in  press  in  J.  Cognitive  Neuroscience. 
Another  theoretical  ms  has  been  submitted  to  Behavioral  &  Brain  Sciences. 


Experiments  involving  Saccade-Contingent  Display  Updating.  One  important 
general  class  of  experiments  involves  changes  in  the  visual  display  during  a  saccade. 
In  the  grant  period  we  have  written  programs  to  do  this  in  the  experimental  set  up 
where  the  head  is  fixed  and  the  eye  monitored  by  the  DPI  eyetracker  and  block 
movements  are  made  using  the  mouse.  Using  these  newly  developed  programs,  we 
have  begun  experiments  which  investigate  three  major  classes  of  questions:  1.  What 
is  the  nature  of  the  visual  information  retained  from  previous  views?  2.  What  are 
the  reference  frames  for  programming  the  various  movements  in  the  task?  3.  What 
is  the  nature  of  the  sub-components  of  the  task. 

Performance  in  the  blocks  task  provides  plausible  evidence  that  subjects  use  fixation 
as  a  deictic  pointing  device  to  serialize  the  task  and  allow  incremental  access  to  the 
immediately  task-relevant  information.  To  test  this,  in  one  experiment  we  changed 
the  color  of  one  of  the  uncopied  blocks  while  the  subject  was  making  a  saccade  to  the 
model  area  following  a  block  placement.  Our  results  so  far  indicate  that  the  target 
selection  involved  in  programming  the  saccade  into  the  model  does  not  involve 
the  acquisition  of  color  information  at  that  location,  and  that  this  function  occurs 
during  the  fixation  in  the  model  area.  This  implies  that  rather  minimal 
information  is  retained  from  the  immediately  prior  fixation,  and  is  consistent  with 
the  suggestion  that  fixation  is  used  for  acquiring  information  just  prior  to  its  use. 
Thus  the  information  retained  from  prior  saccades  is  determined  by  what  is 
currently  relevant  for  the  task. 


Copying  Task  in  a  Natural  Environment 

In  addition,  we  have  investigated  performance  using  real  blocks  and  hand 
movements,  with  the  subjects'  head  free  to  move,  using  an  ASL  head-free  eye  and 
head  tracker,  and  a  magnetic  hand  coil.  (This  equipment  was  bought  on  an  NIH 
Resource  Development  Grant.  Development  of  the  laboratory  was  undertaken  by 
Pelz  and  an  undergraduate  lab  assistant  supported  by  the  grant.  A  new  Mac  to  run 
this  system  will  also  be  purchased  using  AFOSR  funds,  in  order  to  improve  the 
temporal  sampling  rate.)  This  has  provided  important  validation  of  our  task  in 
more  natural  conditions  in  addition  to  revealing  a  number  of  new  findings. 
Subjects  in  the  natural  task  perform  in  the  same  stereotypical  way  as  in  the  Mac 
task,  characterized  by  frequent  eye  movements  to  the  model  pattern.  Thus  the  use  of 
short  term  memory  is  extremely  limited,  even  to  the  extent  that  properties  of  a 
single  block  are  acquired  separately.  Even  when  the  cost  of  references  to  the 
environment  was  increased  by  moving  the  model  and  workspace  70  deg  apart 
(requiring  large  head  movements),  the  subject  still  made  frequent  reference  to  the 
model.  Conversely,  other  experiments  revealed  that  performance  declined 
precipitously  when  frequent  access  to  the  model  during  task  performance  was 
prohibited. 

Subjects  differ  in  the  frequency  with  which  they  make  return  saccades  to  the 
model  but  are  remarkably  similar  in  other  aspects  of  performance,  such  as  fixation 
duration,  time  for  pick  up  and  for  put  down.  In  addition,  the  time  spent  in  the 


model  area  does  not  change  significantly  in  the  course  of  the  trial,  indicating  that 
very  little  learning  of  the  model  takes  place. 

Eye-head  Coordination.  As  well  as  examining  performance  at  the  overall  level  of 
strategies  used,  we  have  also  analyzed  performance  at  the  level  of  the  individual 
actions.  The  coordination  of  the  eyes  and  head  have  been  examined  in  some  detail 
since  it  is  one  in  which  we  can  address  the  question  of  what  kind  of  spatial 
representation  guides  a  change  of  gaze  involving  both  eye  and  head.  A  major  issue 
here  is  whether  the  eye  and  head  are  programmed  with  respect  to  a  shared  spatial 
map.  Current  models  of  changes  in  gaze  postulate  that  both  eye  and  head 
movements  are  driven  by  the  desired  gaze  shift  in  body-centered  or  exocentric 
(spatial)  coordinates  and  that  the  VOR  is  suppressed  until  the  spatial  goal  is 
achieved.  Little  is  known,  however,  about  the  coordination  of  the  eyes  and  head  in 
natural  situations  where  the  head  is  unrestrained  and  sequences  of  goal  directed 
movements  are  involved.  Our  experiments  show  startling  dissociations  of  eye  and 
head  movements,  with  the  eye  and  head  simultaneously  moving  towards  spatially 
separate  goals.  We  are  exploring  these  patterns  and  attempt  to  determine  whether 
these  diverging  movements  are  programmed  independently  and  sequentially,  and 
whether  the  linkage  between  the  movements  is  at  the  highest  level  of  an 
attentional  mechanism  which  flexibly  defines  separate  or  shared  spatial  goals.  Thus 
the  synchrony  of  eye  and  head  may  in  general  be  determined  by  a  combination  of 
attentional  demands  and  the  goal  structure  of  the  task.  Alternatively,  the  saccades 
may  be  programmed  as  a  structured  sequence  of  movements  together  with  the  head 
movement  to  a  common  spatial  goal. 

Development  of  a  Driving  Simulator. 

One  of  the  primary  limitations  of  the  block  copying  task  is  that  the  visual 
environment  is  stationary.  A  convenient  way  to  make  the  visuo-motor  interactions 
time-dependent  is  to  investigate  a  driving  paradigm.  We  are  currently  developing  a 
driving  simulator  using  a  Silicon  Graphics  Reality  Engine  to  generate  scenes 
displayed  on  the  North  Carolina  virtual  reality  LCD  displays,  with  the  capability  of 
tracking  the  eyes  and  head  during  a  simulated  driving  task.  The  display  is  linked  to 
the  head  movements  to  simulate  driving  through  a  landscape.  Currently,  eye 
movements  can  be  recorded,  but  not  used  to  update  the  display.  At  the  moment, 
simulated  motion  through  the  display  is  linked  only  to  accelerator,  brake,  and 
steering  wheel  of  the  go-cart  frame.  One  of  the  primary  goals  of  the  simulator  is  to 
examine  the  nature  of  the  visual  processing  in  unattended  regions  of  the  visual 
field,  and  the  mechanisms  which  govern  attentional  modulation.  Experiments  are 
currently  underway  to  examine  task  sharing  between  following  and  obstacle 
avoidance.  (This  work  has  been  partially  funded  by  an  NIH  Resource  Development 
Grant  as  well  as  the  current  grant.) 


Other  Studies. 


-  Keith  Karn  has  begun  an  investigation  comparing  the  effect  of  external 
reference  frames  on  perception  and  reaching.  Perception  and  action  have  often  been 
thought  to  involve  different  neural  channels.  Karn  is  examining  the  hypothesis 
that  the  differences  are  more  simply  accounted  for  in  terms  of  the  reference  frame 
used  for  the  task  in  question.  These  experiments  will  for  the  basis  for  his 
dissertation,  which  he  plans  to  complete  by  the  end  of  the  year. 

Joel  Lachter  has  continued  experiments  on  how  much  visual  information  gets 
processed  outside  the  focus  of  attention.  He  is  using  a  novel  technique  introduced  by 
Rock  to  look  at  this  question.  He  has  received  his  Ph  D  on  the  basis  of  this  work  and 
is  currently  a  post  doctoral  fellow  at  Princeton.  His  experiments  suggest  that  visual 
processing  outside  the  focus  of  attention  is  minimal  Lachter  and  I  have  also  worked 
on  a  manuscript  on  the  role  of  attention  in  integrating  information  across  saccades. 
This  under  revision  for  Perception  &  Psychophysics 
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